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CODED OBJECT SYSTEM AND CODE RECOGNITION 
METHODS 

BY INVENTORS 

Marcos R. Vescovi and Philippe P. Piernot 

Cross Reference to Related Applications 

This application claims priority to U.S. patent application Ser. No. 
08/675,104 filed July 3, 1995 entitled, "Video Camera Based Computer 
Input System With Interchangeable Physical Interface" by Piernot, et 
al., having assignment rights in common with the present invention, and 
which is herein incorporated by reference. This application is related to 

U.S. Patent application Ser. No. (Attorney Docket No. 

INT 1P0 14) filed concurrently herewith, which is incorporated herein 
by reference for all purposes. 

Background of the Invention 

This invention relates generally to a coded object system and 
computer methods for recognizing codes and more particularly to coded 
objects and code recognition methods within an environment with low 
lighting, low resolution, and low dynamic range capabilities of the code 
reader. 

Bar-codes are a specific type of coded object and are becoming 
increasingly popular as a means of identifying objects. Consequently, 
bar-codes are quickly being introduced into many segments of society. 
That is, bar-codes are not merely being used to identify grocery store 
items in a supermarket. For example, bar-codes are now being used by 
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businesses to keep track of documents and by manufacturers to keep 
track of components within their assembly line. 

When one thinks of conventional bar-codes, one initially thinks of 
bar-codes that are used on product packaging to identify the product 
5 type and price, for example. Typically, a product bar-code is in the 

form of dark lines or "bars" against a light background. The dark bars 
are arranged in parallel and are of varying width. Typically, a bar- 
code reader that includes a laser beam is configured to read the bar- 
code. The width of each bar is determined by calibrating each bar with 
10 a particular frequency of the laser beam. The identification of each 

bar-code is then determined by ascertaining the arrangement of bars and 
associated widths of the bar-code. A similar type arrangement is used 
for recognizing business documents. 

Although the above-described bar-code system works adequately 
1 5 within the business world, this bar-code system has many limitations. 

For example, the bar-code must conform to a set of strict size and shape 
requirements. Also, the bar-code must be positioned directly under the 
laser beam of the bar-code reader or the laser beam must be positioned 
directly over the bar-code. This positioning must be accomplished in 
20 such a way as to allow the laser beam to move perpendicularly across 

the parallel widths of the bars. If these requirements are not met, the 
bar-code reader will fail to correctly identify the bar-code. Thus, one 
must ensure that the bar-codes meet the size and shape requirements, 
and that the bar-code reader is used correctly. Since one cannot 
25 completely eliminate human error, for example, it is difficult to achieve 

perfect reliability for reading and identifying bar-codes in the above- 
described conventional system. 



INT1P012/LVP/MRO 



- 2 - 



PATENT 



Another conventional bar-code system utilizes a camera for 
reading the bar-code, instead of a laser, and is typically found within a 
manufacturing environment. For example, bar-codes are placed on 
components so that a camera may identify each component as the 
components moves through the assembly line. That is, a camera may be 
placed at a specified location such that a bar-code of a component moves 
directly below the camera. The camera translates the bar-code into a 
representational image, and the image is then analyzed and interpreted. 
The camera is typically configured and positioned so that the bar-code 
fills the view of the camera and the camera takes a high-resolution bar- 
code image which is then analyzed and identified. 

Like the conventional laser-based bar-code system, the 
conventional camera-based bar-code system has many disadvantages. 
For example, in order to achieve accurate results, the camera must have 
a high dynamic range, the lighting must be consistently bright, or the 
resolution of the bar-code image must be relatively high. Otherwise, 
the bar-code image may be incorrectly interpreted. For example, under 
low lighting conditions, the bars of the bar-code image may appear 
wider than they actually are. By way of another example, if the 
resolution is too low, the bar-code image may appear blurry. That is, 
one dark area of the bar-code is mapped onto too few pixels, and the 
bar-code image is analyzed incorrectly. In sum, the reliability of the 
camera-based bar-code system is dependent on lighting conditions, 
resolution parameters, and camera specifications. 

As a result of the built-in constraints of the above-described 
conventional bar-codes recognition systems, the above-described 
systems only work in a tightly constrained environment. For example, 
conventional bar-code systems may not work well in the context of a 
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system where the bar-code was positioned within a small area of the 
image. By way of another example, the reliability of conventional bar- 
code systems may be significantly reduced under poor or nonuniform 
lighting conditions. 

5 Thus, there is a need for an improved coded object system and a 

method for recognizing codes within the coded object system even when 
lighting conditions, resolution parameters, and camera capabilities are 
not ideal. That is, what is needed is a method and apparatus for 
recognizing codes within a less constrained environment. Additionally, 
1 0 there is a need for a code recognition system that could be implemented 

in real time and run on a relatively low-end processor. 
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Summary of the Invention 

To achieve the foregoing and other objects and according to the 
purpose of the present invention, a method of tracking coded objects 
that are located within a physical system and are represented within an 
image is disclosed. The method includes the act of receiving a first 
image having a plurality of image pixels that include a first pixel, a first 
neighbor pixel, and a plurality of coded objects pixels that include the 
first pixel, wherein the coded object pixels represent a first coded object 
of the physical system that has a known code pattern and the act of 
obtaining object data about the first coded object by comparing 
luminosity values of the first pixel and the first neighbor pixel. 

In a preferred embodiment, the object data specifies a pixel of the 
plurality of pixels as being a reference pixel within the first coded 
object that has the known code pattern. The act of obtaining the object 
data further includes the acts of comparing luminosity values of the first 
pixel and a first group of neighbor pixels of the plurality of pixels, 
determining a first count of the first group neighbor pixels that have a 
luminosity value that is lower than a first pixel luminosity value of the 
first pixel, determining whether the first count is more than a 
predetermined number, and defining the first pixel as the reference 
pixel of the first code object when the first count is higher than the 
second count. 

In an alternative embodiment, the object data specifies a pixel of 
the plurality of pixels as being a reference pixel within the first coded 
object that has the known code pattern. The act of obtaining object data 
include the acts of comparing luminosity values of the first pixel and a 
first group of neighbor pixels of the plurality of pixels, determining a 
first count of the first group neighbor pixels that have a luminosity 
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value that is lower than a first pixel luminosity value of the first pixel, 
comparing luminosity values of a second pixel and a second group of 
neighbor pixels of the plurality of pixels determining a second count of 
the second group neighbor pixels that have a luminosity value that is 
lower than a second pixel luminosity value of the second pixel, and 
determining whether the second count is more than a predetermined 
number. The act of obtaining object data further includes the act of 
defining the first pixel as the reference pixel of the first code object 
when the first count is higher than the second count and the act of 
defining the second pixel as the reference pixel of the first code object 
when the second count is higher than the first count. 

In another embodiment, the object data indicates a position of the 
first pixel. The act of obtaining the object data further includes the act 
of comparing luminosity values of a first region of pixels of the 
plurality of pixels to determine a minimum value position of a selected 
pixel of the first region that has a minimum luminosity value that is 
lower than any other pixel of the first region, wherein the region 
includes the first pixel, the first pixel having a black color and the other 
pixels of the region having a white color. The act of obtaining the 
object data further includes the act of defining the position of the first 
pixel as being at the minimum value position within the first region. 

In a preferred embodiment, the act of obtaining the object data 
further indicates a position of a second pixel, wherein the first pixel 
being black and the second pixel being white. The act of obtaining the 
object data further includes the act of subtracting the luminosity value 
of each pixel of the plurality of pixels that is positioned within a defined 
white region from an associated pixel of the plurality of pixels that is 
positioned in a defined black region. The pixels of the defined white 
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region are mostly white except for the first pixel, and the pixels of the 
defined black region are mostly black accept for the second pixel. The 
act of obtaining the object data further includes the acts of determining 
a minimum subtraction position for the pixels that result in a lowest 
subtraction result and defining the position of the first pixel and the 
second pixel as being at the minimum subtraction position within the 
white region and the black region. 

In an alternative embodiment, the object data indicates whether 
the first pixel is black or white, and the first neighbor pixel is a known 
white pixel. The act of obtaining the object data includes the acts of 
determining whether a first pixel luminosity value of the first pixel is 
smaller than a first neighbor luminosity value of the first neighbor pixel 
by more than a predetermined value, defining the first pixel as black 
when the first pixel luminosity value of the first pixel is smaller than the 
first neighbor luminosity value of the first neighbor pixel by more than 
a predetermined value, and defining the first pixel as white when the 
first pixel luminosity value of the first pixel is not smaller than the first 
neighbor luminosity value of the first neighbor pixel by more than a 
predetermined value. 

In an alternative embodiment, the object data also indicates 
whether the first pixel is black or white, and the first neighbor pixel is a 
known white pixel. The act of obtaining the object data further includes 
the acts of comparing luminosity values of the first pixel and a second 
neighbor pixel of the plurality of pixels, wherein the first neighbor 
pixel and the second neighbor pixel are known white pixels, and 
determining whether a first pixel luminosity value of the first pixel is 
smaller than a maximum value luminosity of the first neighbor pixel 
and the second neighbor pixel by more than a predetermined value. 
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In an alternative embodiment, the object data also indicates 
whether the first pixel is black or white, and the first neighbor pixel is a 
known white pixel. The act of obtaining the object data further includes 
of the acts of comparing luminosity values of the first pixel and a 
5 second neighbor pixel, wherein the first neighbor pixel is a known 

white pixel and the second neighbor pixel is a known black pixel, 
determining whether the first pixel luminosity value of the first pixel is 
smaller than the first neighbor luminosity value of the first neighbor 
pixel by more than a second neighbor luminosity value of the second 

1 0 neighbor pixel is smaller than the first pixel luminosity value of the first 

pixel, defining the first pixel as black when the first pixel luminosity 
value of the first pixel is smaller than the first neighbor luminosity 
value of the first neighbor pixel by more than the second neighbor 
luminosity value of the second neighbor pixel is smaller than the first 

1 5 pixel luminosity value of the first pixel, and defining the first pixel as 

white when the first pixel luminosity value of the first pixel is not 
smaller than the first neighbor luminosity value of the first neighbor 
pixel by more than the second neighbor luminosity value of the second 
neighbor pixel is smaller than the first pixel luminosity value of the first 

20 pixel. 

In an alternative embodiment, the object data also indicates 
whether the first pixel is black or white, and the first neighbor pixel is a 
known white pixel. The first neighbor pixel has a color that is opposite 
a color of the first pixel, and the act of obtaining the object data further 
25 includes the acts of determining whether a first neighbor luminosity 

value of the first neighbor pixel is greater than a first pixel luminosity 
value of the first pixel, defining the first pixel as black when the first 
neighbor luminosity value of the first neighbor pixel is greater than the 
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first pixel luminosity value of the first pixel, and defining the first pixel 
as white when the first neighbor luminosity value of the first neighbor 
pixel is not greater than the first pixel luminosity value of the first 
pixel. 

In another embodiment of the present invention, a coded object 
that is configured to be scanned by a camera and included in an image 
that is received as input to a computer implemented coded object 
recognition program is disclosed. The coded object includes a first 
region that is represented as a first region image in the received image 
and a second region that is represented as a second region image in the 
received image. The first region has an opposite color of the second 
region such that the coded object may be matched to a known coded 
pattern by determining whether a certain percentage of pixels within the 
second region image have a higher luminosity value than the first region 
image. 
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Brief Description of the Drawing s 
Fig. 1 is a flowchart illustrating a process of recognizing objects 
within a physical system in accordance with one embodiment of the 
present invention. 

5 Fig. 2 is a flowchart illustrating the process of Fig. 1 of tracking 

the coded objects in accordance with one embodiment of the present 
invention. 

Fig. 3 is a flowchart illustrating a single-precision, single-color, 
single- sided method of discriminating pixels in accordance with one 
1 0 embodiment of the present invention. 

Fig. 4 is a flowchart illustrating a single-precision, single-color, 
double-sided method of discriminating pixels in accordance with one 
embodiment of the present invention. 

Fig. 5 is a flowchart illustrating a single-precision, single-color, 
15 combination method (combo method) of discriminating pixels in 

accordance with one embodiment of the present invention. 

Fig. 6 is a flowchart illustrating a single-precision and double- 
color method of discriminating pixels in accordance with one 
embodiment of the present invention. 

20 Fig. 7 is a flowchart illustrating a double-precision method of 

discriminating pixels in accordance with one embodiment of the present 
invention. 

Fig. 8 is a flowchart illustrating a single-precision method of 
detecting the position of a particular object in accordance with one 
25 embodiment of the present invention. 
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Fig. 9 is a flowchart illustrating a double-precision method of 
detecting the position of a particular object in accordance with one 
embodiment of the present invention. 

Fig. 10 is a flowchart illustrating pattern detection and selection 
methods in accordance with one embodiment of the present invention. 

Fig. 1 1 is a coded object in accordance with one embodiment of 
the present invention. 

Fig. 12 is a coded object having a rotating object in accordance 
with an alternative embodiment of the present invention. 

Fig. 13a is a circular coded object in accordance with an 
alternative embodiment of the present invention. 

Fig. 13b is a circular coded object having a double-precision 
angle point in accordance with an alternative embodiment of the present 
invention. 

Fig. 13d is a circular coded object having a single-precision angle 
point and double-precision bits in accordance with an alternative 
embodiment of the present invention. 

Fig. 13e is a circular coded object having a double-precision 
angle point and single-precision bits in accordance with an alternative 
embodiment of the present invention. 

Fig. 14 is a coded object having a double-precision movable 
object in accordance with an alternative embodiment of the present 
invention. 

Fig. 15 is a flowchart illustrating the process of Fig. 1 of tracking 
the non-coded objects in accordance with one embodiment of the present 
invention. 
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Fig. 16 is a diagrammatic representation of two successive frames 
having multiple non-coded objects that are to be matched in accordance 
with one embodiment of the present invention. 
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Detailed Description of the Preferred Embodiments 

While this invention has been described in terms of several 
embodiments and specific examples, there are alterations, permutations, 
and equivalents which fall within the scope of this invention. It is 
5 therefore intended that the following appended claims be interpreted as 

including all such alterations, permutations, and equivalents as fall 
within the true spirit and scope of the present invention. 

In general terms, the current invention includes a method and 
apparatus for analyzing an image having a plurality of coded physical 

1 0 objects, and recognizing and analyzing the coded physical objects under 

variable conditions. The present invention may be implemented for 
tracking coded objects that have fixed locations, as well as tracking 
coded objects that have variable locations. (The term "tracking" includes 
the acts of recognizing, interpreting, and/or analyzing coded objects 

15 within a particular frame, as well as from frame to frame). 

The present invention may be implemented on any conventional 
hardware and computer interface system. For example, the present 
invention is well suited for recognizing and tracking game pieces within 
a physical game system, where the user may manipulate physical objects 
20 or game pieces on the game board, or toy board. The present invention 

is especially well suited for tracking coded objects (or game pieces) in a 
physical game system. 

Basically, an image is received from a camera that has obtained 
the image of the physical system, including the coded objects. The 
25 image is analyzed using techniques, which are described below, to track 

the coded object. For example, the techniques of the present invention 
may be implemented to identify and interpret the coded objects. 
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Although portions of the following description are in reference to 
a game board, of course, it should be well understood by those skilled in 
the art that the application of the current invention is not limited to a 
game board, but may be implemented with any appropriate physical 
5 objects where recognition of coded objects is required. Although the 

following description is in reference to a computer system interface, it 
should be understood that other type of interfaces may be used, such as 
a television game system or set top box. 

The present invention may be implemented with a video camera 

10 computer input system having an interchangeable spatial reference 

member with various physical objects thereon. One example of an 
interchangeable spatial reference member is a game board. A camera is 
positioned to obtain an image of the interchangeable spatial reference 
member (or game board) and physical objects (game pieces). Several 

15 examples of various video camera computer input systems and spatial 

reference members are described in U.S. patent application Ser. No. 
08/675,104 filed July 3, 1995 entitled, "Video Camera Based Computer 
Input System With Interchangeable Physical Interface" by Piernot, et 
al., and which is herein incorporated by reference. 

20 A computer method may be provided for interfacing with the 

physical objects of the game board in a complex manner. As the user 
interacts with the physical objects, the user is provided with a 
corresponding audiovisual interaction on the computer, which 
interaction greatly enhances the game experience. Several examples of 

25 computer interfaces for a physical game board are described in co- 
pending U.S. Patent application Ser. No. (Attorney Docket 

No. INT1P014) filed concurrently herewith, which is incorporated 
herein by reference for all purposes. 
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Fig. 1 is a flowchart illustrating a process 100 of recognizing 
objects within a physical system in accordance with one embodiment of 
the present invention. Initially, in operation 102, image data is 
received. This image may include a representation of the entire 
underside of the board game, for example. That is, the image will 
include coded objects that are associated with particular physical objects 
on the game board. The image may also include non-coded objects, 
such as the user's hand as detected through a transparent board, for 
example. 

After the image data is received, the camera exposure may be 
adjusted to obtain the best image in operation 104. Of course, the 
camera may not have to be adjusted if the image is already optimal or 
object identification reliability is at an acceptable level. In operation 
106, the coded objects are tracked, which operation is further described 
in reference to Fig. 2 through 10. In operation 108, the non-coded 
objects are tracked. Of course, operation 108 is optional and is 
dependent on the application and whether it is necessary to track non- 
coded objects. Operation 108 is described in more detail below with 
reference to Figs. 15 and 16. 

After the non-coded objects, if any, are tracked, object data that 
was obtained during the tracking operations 106 and 108 is stored in 
operation 110. Object data includes information that may be utilized to 
identify, interpret, and/or recognize the objects or portions of the coded 
object. Next, operations 102 through 110 are repeated so that another 
set of image data is received, the camera is adjusted, coded and non- 
coded objects are tracked, a new set of object data is stored, and 
operations 102 through 110 are repeated for another received image. 
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The adjusting procedure of operation 104 may be accomplished 
by a conventional luminosity-based method. For example, the exposure 
may be adjusted such that the average luminosity of the entire image is 
50%. However, this conventional exposure adjustment method is 
dependent on the objects of the physical system. For example, if the 
physical system has an unusual number of dark objects, the exposure 
may be incorrectly adjusted such that the image is overexposed. 

Alternatively, a more preferred embodiment of the present 
invention includes a method for adjusting the exposure by analyzing the 
entropy level of the image. That is, an entropy level of the image is 
computed. An exposure time is then computed that will be expected to 
maximize the entropy of the image. In one embodiment, a PID control 
algorithm is used. The entropy of an image represents the amount of 
information contained in the image, as is well understood by those of 
skill in the art of information processing. Maximizing the amount of 
information contained in the image is one way that an optimal exposure 
time may be determined. Although this method works well for objects 
on sealed opaque surface, this method also has the disadvantage of being 
dependent on the objects in the physical system. 

In a most preferred embodiment, the physical system includes a 
tag that is designed to have a predetermined luminosity. The exposure 
is adjusted so that the luminosity of the tag is equal to the predetermined 
luminosity. For example, if the tag is designed to have a middle-gray 
color, the exposure will be adjusted such that the tag has a 50% 
luminosity. This tag-based adjustment method has the advantage of 
being dependent only on the lighting of the system and not on the 
objects of the physical system. 
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Fig. 2 is a flowchart illustrating the process 106 of Fig. 1 of 
tracking the coded objects in accordance with one embodiment of the 
present invention. In contrast to conventional thresholding and 
segmentation techniques that are used for tracking non-coded objects, 
5 the process 106 of tracking coded objects works with the gray-image. 

Tracking coded objects is accomplished by comparing pixel 
luminosity values within the image. The results of the comparison are 
analyzed in various ways and are used to recognize and interpret the 
coded objects by matching a pixel to a reference pixel of a known code 
10 pattern (operation 202), discriminating pixels (operation 204), and/or 

determining positions of particular objects relative to a known region 
(operation 206). 

Of course, it may not be necessary to perform all four operations 
in order to track a coded object. The inclusion or exclusion of 
1 5 operations 202 through 206 depends on the design of the coded object 

and requirements of the application in which the coded object is being 
used. Operation 202 through 206 may be executed in any order that is 
suitable for tracking a particular coded object. 

Additionally, each operation that is used for tracking coded 
20 objects has several alternative embodiments (which are described below 

in reference to Figs. 3 through 10), where the embodiment choice 
depends on the particular design of the coded object and application 
being used. Several alternative embodiments of coded objects that work 
well with one or more of the alternative embodiments for tracking 
25 coded objects are described below in reference to Figs. 11 through 13. 

Although specific techniques have been described for tracking coded 
objects in reference to Figs. 3 through 10, these techniques are merely 
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example techniques for tracking coded objects by comparing pixel 
luminosity values and are not meant to limit the scope of the invention. 

Turning now to the process 106 of Fig. 2 of tracking coded 
objects, a particular pixel is matched with a reference pixel in a known 
5 code pattern in operation 202. For example, it is determined which 

pixel matches a center pixel of the known code pattern. In operation 
204, pixels are discriminated. That is, it is determined whether a 
particular pixel (herein after referred to as "pixel X") is a black pixel 
or a white pixel. In operation 206, positions of particular objects are 
10 determined within a defined region of the image. For example, a 

particular object may be found within a defined region by implementing 
the techniques of operation 206. 

Operations 202 through 206 are described below in detail in 
reference to Figs. 3 through 10. Specifically, Figs. 3 through 7 are 

1 5 flowcharts illustrating methods of discriminating pixels (operation 204) 

in accordance with various embodiments of the present invention. Figs. 
8 and 9 are flowcharts illustrating methods of detecting the positions of 
objects (operation 206) in accordance with two alternative embodiments 
of the present invention. Fig. 10 is a flowchart illustrating a method of 

20 detecting and selecting a known code pattern (operation 202) in 

accordance with one embodiment of the present invention. 

Fig. 3 is a flowchart illustrating a single-precision, single-color, 
single-sided method of discriminating pixels in accordance with one 
embodiment of the present invention. Specifically, Fig. 3 illustrates a 
25 method of determining whether pixel X, for example, is a black pixel or 

a white pixel by comparing and analyzing luminosity values of pixel X 
and a neighbor pixel Z, which is a known white pixel The following 
procedure works under the assumption that if pixel X is black, its 
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luminosity value will be lower than neighbor pixel Z's by at least a 
certain amount. 

Initially, in operation 302 pixel X is compared to the neighbor 
pixel Z, which is a known white pixel. Next, in operation 304 it is 
determined whether the luminosity value of pixel X is smaller than the 
luminosity value of pixel Z by more than a predetermined constant. 
Preferably, the predetermined constant is greater than zero so as to 
increase the likelihood that a white pixel under nonuniform lighting 
conditions will be characterized appropriately. Most preferably, the 
predetermined constant is set to a medium value, such as 50% 
luminosity, such that the predetermined constant is greater than the 
maximum margin between two white pixels Z and X under nonuniform 
lighting conditions, for example. 

Since pixel Z is a white pixel, pixel X will be substantially smaller 
than pixel Z if pixel X is black. (Herein after, the term "smaller" means 
that a luminosity value of a pixel is smaller than another pixel's 
luminosity value). However, if pixel X is white, pixel X will be about 
equal to pixel Z. Thus, in this embodiment, the color of pixel X is 
ascertained by comparing the values of pixel X and a white pixel Z. 
That is, when pixel X is smaller that pixel Z by more than the 
predetermined constant, pixel X is defined as a black in operation 306. 
When the pixel X is not smaller than pixel Z by more than a 
predetermined constant, pixel X is defined as white in operation 308. 

This embodiment may be implemented, for example, by the 
following expression: 

IF (pixel Z- pixel X) > k 
THEN pixel X = black 
ELSE pixel X = white 
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ENDIF. 



The constant "k" is the predetermined constant, which value is chosen 
such that accuracy is optimized for variable lighting conditions and 
5 corresponding variable luminosity values for a white pixel Z and X. 

Although the neighbor pixel Z has been described in this 
embodiment as being a white pixel, it should be realized by those skilled 
in the art that pixel Z may be black with only slight modifications to the 
operations of process 106. For example, operation 304 would, for 

1 0 example, be modified to determine whether pixel Z is smaller than pixel 

X by more than a predetermined constant. Alternatively, operation 304 
may be modified to determine whether pixel X is larger than pixel Z; 
operation 306 would be modified to define pixel X as a white pixel, for 
example, when pixel X was found to be larger. In sum, there are 

15 several techniques that may be used for discriminating pixel X by 

comparing luminosity values of pixel X and a neighbor pixel that 
represents an object in the physical system that has a known color. 

Fig 4 is a flowchart illustrating a single-precision, single-color, 
double-sided method of discriminating pixels in accordance with one 

20 embodiment of the present invention. In this embodiment, luminosity 

values are compared and analyzed for pixels X, Z, and W. Pixels Z and 
W are known white pixels. The following procedure works under the 
assumption that if pixel X is black, it will have a luminosity value that is 
lower than the brightest neighbor pixel. In other words, at least one of 

25 pixel Z or W will be significantly brighter than pixel X if pixel X is, in 

fact, black. 

Initially, in operation 402 a maximum value of neighbor pixels Z 
and W is determined. In operation 404, this maximum value is 
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compared to the value of pixel X. Preferably, pixels W and Z are on 
opposite sides of pixel X. This opposite placement is to increase the 
likelihood that one of pixels W and Z will be exposed to the same 
amount of light or more as pixel X. Alternatively, pixel X may be 
compared to white neighbor pixels that are placed on all four sides 
adjacent to pixel X; this "quad-sided" method would improve reliability 
of the discrimination process. 

Next, in operation 406 it is determined whether the value of pixel 
X is smaller than the maximum value by more than a predetermined 
value. When pixel X is smaller than the maximum value, pixel X is 
defined as black in operation 410. When pixel X is not smaller than the 
maximum value by the predetermined value, pixel X is defined as white 
in operation 408. 

Preferably, the predetermined value is greater than zero to 
increase the likelihood that a white pixel X will be characterized 
appropriately. Most preferably, the predetermined value is set to a 
medium value, such as 50% luminosity. The predetermined value may 
be greater than the predetermined constant used in the single-sided 
method of Fig. 3. That is, the difference between the maximum value 
and the value of pixel X in the double-sided method of Fig. 4 is likely to 
be greater than the difference between pixel X and pixel Z in the single- 
sided method of Fig. 3. 

This embodiment may be implemented, for example, by the 
following expression: 

IF (Max(pixel Z, pixel W) - pixel X) > k 
THEN pixel X = black 
ELSE pixel X = white 
ENDIF. 
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The function "Max(pixel Z, pixel W)" returns the maximum value of 
pixel Z and W. The constant "k" represents the predetermined value, 
which value is chosen such that accuracy is optimized for the double- 
sided method. 

The double- sided method of Fig 4 is more reliable than the 
single-sided method of Fig. 3 since two neighbors are used to 
determined the color of pixel X, rather than one neighbor. This double 
comparison decreases the likelihood that pixel X will be compared to an 
unusually dark white neighbor, which may result in erroneously 
defining a black pixel X as white. However, the double-sided method is 
less convenient than the single-sided method since the coded object may 
be larger in size to accommodate the extra neighbor pixel and an extra 
comparison operation is required. 

Fig 5 is a flowchart illustrating a single-precision, single-color, 
combination method (combo method) of discriminating pixels in 
accordance with one embodiment of the present invention. This combo 
method combines the methods of Fig 3 and Fig 4, and is more accurate 
and thereby is preferable to the methods of Fig 3 and Fig 4 of 
discriminating pixels. In this embodiment, for pixel X to be defined as 
black, pixel X has to be darker than the brightest white neighbor pixel 
by a predetermined value and at least darker than the lesser bright 
neighbor. 

Initially, in operation 502 a maximum value of neighbor pixel Z 
and neighbor pixel W is determined. In this embodiment, pixel Z and 
pixel W are known white pixels. Next, in operation 504 a minimum 
value of pixel Z and pixel W is determined. In operation 506 a value of 
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pixel X is compared to both the maximum value and the minimum 
value. 

In operation 508, it is determined whether the value of pixel X is 
smaller than the maximum value by more than a predetermined value. 
5 If the value of pixel X is not smaller than the maximum value by more 

than a predetermined value, pixel X is defined as white in operation 
512. If the value of pixel X is smaller than the maximum value by 
more than a predetermined value, it is determined whether the value of 
pixel X is smaller than the minimum value by more than zero in 
10 operation 510. If the value of pixel X is not smaller the minimum value 

by more than zero, pixel X is defined as white in operation 512. If the 
value of pixel X is smaller than the minimum value by more than zero, 
pixel X is defined as black in operation 514. 

This embodiment may be implemented, for example, by the 
1 5 following expression: 

IF ((Min(pixel Z, pixel W) - pixel X) > 0) & ((Max(pixel Z, pixel W)) - pixel 

X)>k) 

THEN pixel X = black 
ELSE pixel X = white 
20 ENDIF. 



The function "Min(pixel Z, pixel W)" returns the minimum value (or 
brightest) of pixel Z and W, and the function "Max(pixel Z, pixel W)" 
returns the maximum value of pixel Z and W. The constant "k" 
25 represents the predetermined value, which value is chosen such that 

accuracy is optimized for the combo method. 

Fig 6 is a flowchart illustrating a single-precision and double- 
color method of discriminating pixels in accordance with one 
embodiment of the present invention. In general, this method compares 



INT1P012/LVP/MRO 



- 23 - 



PATENT 



an unknown pixel X to a neighbor that is know to be white and a 
neighbor that is known to be black. If pixel X is closer in color to the 
black neighbor, it is defined as a black pixel. If pixel X is closer in 
color to the white pixel, it is defined as a white pixel. The following 
techniques are one example of how the results of these comparisons may 
be analyzed. 

Initially, in operation 602 a value of pixel X is compared to a 
value of neighbor pixel Z. Pixel Z is a known white neighbor pixel. In 
operation 604, the value of pixel X is next compared to a value of 
neighbor pixel Y, which pixel Y has a known black color. After pixel 
X is compared to both neighbors, in operation 606 it is determined 
whether the difference between the values of pixels Z and X is greater 
than the difference between the values of pixels X and Y. 

If the difference between the values of pixel Z and pixel X are 
greater than the difference between the values of pixel X and pixel Y, 
pixel X is defined as black in operation 610. If the difference between 
the values of pixel Z and pixel X are not greater than the difference 
between the values of pixels X and pixel Y, pixel X is defined as white 
in operation 608. This embodiment may be implemented, for example, 
by the following expression: 

IF (pixel Z - pixel X) > (pixel X - pixel Y) 
THEN pixel X = black 
ELSE pixel X = white 
ENDIF. 

Fig 7 is a flowchart illustrating a double-precision method of 
discriminating a pixel in accordance with one embodiment of the 
present invention. For this method, coded objects are designed such that 
when pixel X is black then a neighbor pixel Z is white, and when pixel 
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X is white then pixel Z is black. Thus, to determine the color of a pair 
of pixels that are designed for the double-precision method, each pixel 
is compared to the other pixel with the lower value pixel being defined 
as black and the higher value pixel being defined as white. Examples of 
5 types of coded objects, which may be used with this double-precision 

method, are described below with references to Fig 12, for example. 

Initially, in operation 702 the value of pixel X is compared to the 
value of neighbor pixel Z. Pixel Z may be either black or white. Next, 
in operation 704 it is determined whether the value of pixel Z is greater 
1 0 than the value of pixel X. If pixel Z is greater than pixel X, pixel X is 

defined as black (pixel Z is white) in operation 706. If pixel Z is not 
greater than pixel X, pixel X is defined as white (pixel Z is black) in 
operation 708. 

This embodiment may be implemented, for example, by the 
1 5 following expression: 

IF (pixel Z) > (pixel X) 

THEN pixel X = black and pixel Z = white 

ELSE pixel X = white and pixel Z = black 

ENDIF. 

20 Alternatively this embodiment may be implemented, for example, 

by the following expression: 

IF (pixel Z- pixel X) > k 

THEN pixel X = black and pixel Z = white 

ELSEIF (pixel X- pixel Z) > k pixel X = white and pixel Z = black 
25 ENDIF. 

The constant "k" is a predetermined constant, which value is chosen 
such that accuracy is optimized for the double-precision method. For 
most physical systems, the constant k may be set to zero, since a white 
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pixel usually has a higher luminosity value than a black pixel under 
normal lighting conditions. However, the constant k may have to be set 
to a value other than zero under certain poor lighting conditions. 

Fig. 8 is a flowchart illustrating a single-precision method of 
detecting the position of a particular object of the coded object in 
accordance with one embodiment of the present invention. Initially, in 
operation 802 all pixels in a region are compared. All pixels within the 
region, except pixel X have a particular color, and pixel X has an 
opposite color from the other pixels in the region. For example, if 
pixel X is black, the other pixels within the region are white. This 
difference in colors may be utilized to determine the position of pixel 
X. Pixel X is associated with the particular object; thus, the position of 
the particular object may be determined from the position of pixel X. 
Although the following method is described as determining the position 
of pixel X, of course, the position of a group of pixels (group X) may 
be determined so as to determine the position of the particular object by 
implementing a similar method. 

In operation 804, it is determined whether the region includes 
mostly white or black pixels. Of course, this operation 804 is not 
necessary when the color of the region is already known. If the region 
is mostly black, in operation 806 a maximum value within the region is 
determined, and the position of pixel X is defined in operation 808 as 
the position of the pixel having the maximum value. However, if the 
region is mostly white, in operation 810 a minimum value within the 
region is determined, and the position of pixel X is defined in operation 
812 as the position of the pixel having the minimum value. 

This embodiment may be implemented, for example, by the 
following expression: 
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IF (most region pixels = black) 

THEN position of pixel X = position of Max(pixels within the region) 
ENDIF 

IF (most region pixels = white) 
5 THEN position of pixel X = position of Min(pixels within the region) 

ENDIF. 

Of course, those skilled in the art would realize that other expressions 
may be utilized for the single-precision positioning method. 

10 Fig 9 is a flowchart illustrating a double-precision method of 

detecting the position of a particular object of the coded object in 
accordance with one embodiment of the present invention. For 
example, the particular object may be movable within a region that 
includes a white neighbor pixel and a black neighbor pixel, the white 

1 5 neighbor pixel changes to a black pixel as the particular object moves 

over the white neighbor pixel. Likewise, the black neighbor pixel 
changes to white. Examples of this type of coded object, which may be 
used with this double-precision method, are described below with 
references to Fig. 14, for example. 

20 Initially, in operation 902 all pixels of a first half of a region are 

compared with associated pixels of a second half of the region. In this 
example, the first half pixels are mostly black (pixels Z), and the second 
half pixels are mostly white (pixels Y). Next, in operation 904 a 
difference between each pair of pixels Z and Y is determined. In 

25 operation 906, a position of the particular object is defined as the pair 

of pixels Z and Y that has the minimum difference. 

This embodiment may be implemented, for example, by the 
following expression: 

IF ((pixels Z = black) & (pixels Y = white) 
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THEN position of pixel X = Min(pixel Z - pixel Y) 
ENDIF. 

Of course, those skilled in the art would realize that other expressions 
may be utilized for the double-precision positioning method. 

Fig 10 is a flowchart illustrating pattern detection and selection 
methods in accordance with one embodiment of the present invention. 
These methods are used to identify a particular pixel as being a possible 
reference pixel of a known code pattern. Additionally, these methods 
are used for determining which of the possible pixels (if more than one) 
more closely matches the reference pixel of the known code pattern. 

Initially, in operation 1002 a region having an area that is equal 
to a known bar code pattern area is defined. Next, in operation 1004 an 
expected black pixel is compared to each expected white pixel. In 
operation 1006, it is determined whether more than a predetermined 
number of expected white pixels have a value that is greater than a value 
of the expected black pixel. If the number is higher than the 
predetermined number, in operation 1008 the expected black pixel is 
defined as being a possible reference pixel of the known code pattern. 

After the expected black pixel is defined as a possible reference 
pixel of the known code pattern, in operation 1010 it is determined 
whether more than one expected black pixel has been defined as a 
possible reference pixel of the known code pattern. If more than one 
possible reference pixel has been defined, in operation 1012 the 
expected black pixel that has the highest number of expected white 
pixels that have a value that is greater than a value of the expected black 
pixel is defined as the correct reference pixel of the known code 
pattern. The process ends in operation 1018. 
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An alternative technique may include determining whether a sum 
of each luminosity value of each expected white pixel minus the 
luminosity value of the expected white pixel is greater than a 
predetermined value. Additionally, this embodiment may be utilized 
5 with minor changes to determine which white pixel is equivalent to the 

reference pixel. In this example, an expected white pixel is compared to 
a group of expected black pixels. 

If it is determined in operation 1006 that a predetermined number 
of expected white pixels do not have a value that is greater than a value 

10 of the expected black pixel, the process proceeds to operation 1014 and 

another region is analyzed. In operation 1014, it is determined whether 
all the regions have been analyzed. If all the regions have not been 
analyzed, the process proceeds to operation 1002. If all the regions 
have been analyzed, it is determined that the reference pixel has not 

15 been found within the image in operation 1016. Consequently, the 

process ends in operation 1018. 

The above described tracking methods may be implemented on 
various types of coded objects. For example, a coded object may 
include a reference object that is configured to take advantage of a 

20 double-precision pattern positioning method embodiment. A reference 

object that utilizes a double-precision detection embodiment will be 
recognized more accurately than a reference object that utilizes a single- 
precision positioning embodiment. The following coded objects include 
several features that are designed to utilize various tracking 

25 embodiments. 

Additionally, the above-described tracking embodiments may be 
implemented to track coded objects that have a fixed position, as well as 
objects that are movable within the physical system or within a fixed 



INT1P012/LVP/MRO 



- 29 - 



PATENT 



region of the physical system. Several embodiments of fixed coded 
objects and movable coded objects are described in co-pending U.S. 

Patent application Ser. No. (Attorney Docket No. 

INT1P014) filed concurrently herewith, which is incorporated herein 
5 by reference for all purposes. 

Fig. 11 is a coded object 1100 in accordance with one 
embodiment of the present invention. The coded object 1100 includes a 
plurality of bits 1102, a reference pixel 1104, and a center pixel 1106. 
The center pixel 1106 may be used to detect whether the coded object 
10 1 100 is present. The bits 1 102 may be used to identify the coded object 

1100. The reference pixel 1104, along with the center pixel 1106, may 
be used to detect the orientation of the coded object. 

Of course, any suitable configurations may be used for the center 
pixel 1106, bits 1102, and reference pixel 1104, as long as the coded 
15 object may be tracked using the tracking methods of the present 

invention. For example, the center pixel may not be positioned within 
the center of the coded object 1100. 

The coded object 1100 is configured such that a variation of the 
combo single-precision method of Fig. 5 may be implemented to detect 

20 the presence of the coded object. The pixels that are positioned along a 

horizontal axis 1108 are analyzed, as well as along a vertical axis 1104. 
For this type of coded object, a ""valley" is expected in the luminosity 
values of the pixels along one of the axis. That is, the value of the 
center pixel 1106 will be at a lower value than any of the other pixels 

25 along the horizontal axis 1108. The pixels of the coded object 1100 are 

analyzed for both the horizontal and vertical axis to increase the 
reliability of this method. 
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The bits 1102 are designed to utilize the double-precision 
discrimination method of Fig. 7. That is, each bit includes a black pixel 
and a white pixel. For example, bit 1 102b has a black pixel 1 103 a and a 
white pixel 1103b. The value of the bit 1102b may be determined by 
comparing pixel 1103a to pixel 1103b. The lowest value pixel is the 
black pixel, and the highest value pixel is the white pixel. The relative 
colors of each pixel 1103a and 1103b indicate the value of the bit 1102. 
That is, different coded objects may have different relative colors and 
corresponding values for these bits 1102. 

Fig. 12 is a coded object 1200 having a rotating object 1212 in 
accordance with an alternative embodiment of the present invention. In 
other words, the coded object has a rotating object 1212 that may be 
rotated by the user, for example. The position of the rotating object 
1212 may be determined by implementing the single-precision method 
of Fig. 8 of detecting a position of a black pixel within a mostly white 
region. In other words, a minimum value pixel is found within a circle 
1214 of mostly white pixels. The position of the rotating object 1212 is 
defined as the position of a minimum value pixel within the circle 1214. 

The coded object 1200 also includes a code detection object 1202. 
As shown, the code detection mechanism includes a black object 1216a 
beside a white neighbor object 1216b. When the coded object is added 
to the system, the black object 1216a moves over the white neighbor 
object 1216b and another white neighbor appears to the left of the black 
object 1216a (see 1102d of Fig. 11). Thus, one may determine whether 
the coded object 1200 is present by comparing the luminosity values of 
the black object 1216a and the white object 1216b. 

Fig. 13a is a circular coded object 1302 in accordance with an 
alternative embodiment of the present invention. The circular coded 
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object 1302 has a white center 1306, a black ring 1304, and a white ring 
1308. To detect this code pattern 1302, an embodiment of the pattern 
detection and selection methods of Fig. 10 are implemented. For 
example, a pixel in the center 1306 and pixels in the white ring 1308 are 
compared to pixels in the black ring 1304. The coded object 1302 is 
detected when a particular number of pixels within the black ring 1304 
are darker than pixels in the white ring 1308 and white center 1306. 
Additionally, more than one center pixel 1306 may be compared with 
each of the pixels in the white ring 1308 and black ring 1304, which 
additionally comparison would increase the reliability of detecting the 
correct coded object. 

Fig. 13b is a circular coded object 1310 having a single-precision 
angle point 1312 in accordance with an alternative embodiment of the 
present invention. The single-precision angle point 1312 may be 
utilized to determine the orientation of the coded object 1310, as well as 
bit locations and values within the coded object 1310. Examples of 
different types of code bits are described in reference to Figs. 13c 
through 13e. 

An embodiment of the single-precision position detection method 
is implemented to detect the position of the angle point 1312 within a 
region 1316. As shown, the region 1316 includes mostly black pixels. 
The black pixels and the angle point 1312 form a ring around the 
perimeter of the code 1310. The angle point 1312 is defined as the 
lightest pixel in the region 1316. 

Fig. 13c is a circular coded object 1320 having a double-precision 
angle point 1322 in accordance with an alternative embodiment of the 
present invention. The double-precision angle point 1322 includes a 
white pixel 1322b that is located within a black ring region 1324b, and a 
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black pixel 1322a that is located in within a white ring region 1324a. 
The coded object 1320 is more robust than the coded object 1310 having 
the single-precision angle point. The double-precision angle point 1322 
increases the likelihood that the angle point 1322 will be detected 
5 correctly. 

The position of the angle point is determined by comparing the 
pixels in the white ring region 1324a to adjacent pixels in the black ring 
region 1324b. That is, a double-precision positioning method is used 
that is described in reference to Fig. 9. For example, a pixel in the 

10 white ring region 1324a is subtracted from an adjacent pixel in the 

black ring region 1324b, wherein each pair is located along a radial line 
that goes through a center of the coded object 1320 at a particular angle. 
This calculation is repeated for each pair of pixels within the white ring 
region 1324a and the black ring region 1324b. The position of the 

15 angle point 1322 is defined as the pair of pixels that result in a 

minimum subtraction value. In other words, the subtraction value for 
the angle point 1322 ( a black pixel minus a white pixel) will be less 
than for all other pixel pairs within the ring regions (a white pixel 
minus a black pixel). 

20 Fig. 13d is a circular coded object 1330 having a single-precision 

angle point 1332 and double-precision bits 1334 in accordance with an 
alternative embodiment of the present invention. Each of the double- 
precision bits 1334 include a white region and a black region. For 
example, a first bit is in the form of a white region 1334a and a black 

25 region 1334b. The bits are used to identify the coded object type, for 

example, with each particular coded object type having a different bit 
value arrangement. 
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The value of each bit is determined by reading the value of each 
subregion within the bit (e.g., 1334a and 1334b). The double-precision 
discrimination method may be used to determine the value of each bit. 
In other words, each bit includes a white pixel that is adjacent to a black 
pixel, the color of each pixel may be ascertained by comparing 
luminosity values of pixels 1334a and 1334b. 

Fig. 13e is a circular coded object 1340 having a double-precision 
angle point 1342 and single-precision bits 1344 in accordance with an 
alternative embodiment of the present invention. This embodiment may 
result in a smaller code pattern size than the coded object 1330 having 
the double-precision bits since the single-precision bits of coded object 
1340 requires one less ring region. 

In this embodiment, one of the single-precision discrimination 
methods may be utilized for determining the value of each bit 1344 of 
code pattern 1340. For example, the color of each bit 1344 may be 
determined by utilizing the single-precision, double-color 
discrimination method of Fig. 6. Each bit is compared to a white 
neighbor pixel and a black neighbor pixel. 

The angle point 1342 may be utilized as white and black neighbor 
pixels, for example. First, the position of the angle point 1342 is 
determined by utilizing the double-precision positioning method of Fig. 
9, for example. That is, the lightest pixel 1342b within a black ring 
region 1346 is defined as the white neighbor pixel, and the darkest pixel 
within the a white ring region 1348 is defined as the black neighbor 
pixel. After the neighbor pixels are determined, each bit is compared to 
both the white pixel 1342b and the black pixel 1342a in a double-color 
discrimination method of Fig. 6 to determine the value of each bit. 
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Fig. 14 is a coded object 1400 that includes a double-precision 
movable object that includes a white pixel 1402b and a black pixel 
1402a in accordance with an alternative embodiment of the present 
invention. The coded object 1400 includes a mostly white half 1402 and 
5 a mostly black half 1404. The white pixel 1402b of the movable object 

is part of the mostly black half 1402. Likewise, the black pixel 1404b 
of the movable object is part of the mostly white half 1404. 

The movable object (1402a and 1404a) is slidably movable within 
the halves 1402 and 1404. For example, a user may move the movable 
1 0 object from the left side of the halves to the right side. As the movable 

object passes through each half, the pixels of each half change colors. 
In other words, each of the movable object pixels (1402b and 1404b) 
has a color that is a reverse of most pixels within the associated half. 

The coded object 1400 of Fig. 14 is one example of a coded 
1 5 object that may utilize the double-precision positioning method of Fig. 9 

to determine the position of the movable object. For example, each 
pixel in the mostly white half 1404 is subtracted from each pixel in the 
mostly black half 1402. The position of the pair of pixels that result in 
a minimum value are defined as the position of the movable object 
20 (1402b and 1402a). 

The coded object recognition system of the present invention has 
many advantages over conventional bar-code recognition systems. If 
one wanted to track and identify many coded physical objects within a 
physical system, such as an interactive game board, the image resolution 
25 and lighting conditions would sometimes be inadequate for conventional 

code recognition systems to function properly. That is, since the 
interactive game board is designed for home use, the lighting conditions 
will radically vary. 
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For example, the lighting conditions could result in few levels of 
gray, which would make it difficult to distinguish black from white and 
thereby to recognize whether a code is present. Additionally, 
conventional code systems would require an expensive camera that has a 
high dynamic range in order to adjust for inadequate lighting 
conditions, for example. Since an image of the physical system may 
contain many coded objects, the resolution for each coded object may be 
relatively low. In contrast to conventional systems, the present 
invention will recognize codes even, for example, with variable lighting 
conditions, low resolution parameters, and low dynamic range cameras. 

Additionally, since the code recognition techniques of the present 
invention include relatively simple calculations, the present invention 
may be implemented on a relatively low-end, low-cost processor. That 
is, the techniques of the present invention minimize the number of 
floating point operations that are required to track coded objects. Thus, 
the present invention may also be implemented in real time for a game 
application, for example. 

Fig. 15 is a flowchart illustrating the process 108 of Fig. 1 of 
tracking the non-coded objects in accordance with one embodiment of 
the present invention. Process 108 may be eliminated if no non-coded 
objects are present or if it is not desirable to track non-coded objects. 
However, in some applications, it may be desirable to track non-coded 
objects, such as the user's hand. 

Initially, in operation 1501, all non-coded objects are located 
within each image or frame. This location operation is accomplished by 
conventional object location techniques. A well known method for 
locating non-coded objects within an image is to utilize thresholding and 
segmentation techniques. Additional object location techniques are 
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described in co-pending U.S. patent application Ser. No. 08/675,104 
filed July 3, 1995 entitled, "Video Camera Based Computer Input 
System With Interchangeable Physical Interface" by Piernot, et al., and 
which is herein incorporated by reference. 

After the objects are located for each frame, the objects are 
matched or linked in successive frames in operations 1502 through 
1508. Fig. 16 is a diagrammatic representation of two successive 
frames having multiple non-coded objects that are to be matched in 
accordance with one embodiment of the present invention. Fig. 16 will 
be described in conjunction with the flowchart of Fig. 15. In general, 
particular objects are identified and distinguished from all other objects 
in each successive frame so that each object's movement may be 
accurately tracked from frame to frame. 

In Fig. 16, a first frame 1602 has a first object 1608 and a second 
object 1610. In a second frame 1604, the objects 1608 and 1610 have 
moved to one of two positions 1612 and 1614. For example, object 
1612 of the second frame 1604 is either object 1608 or 1610 of the first 
frame 1602. The second frame is represented by two possible 
interpretations, 1604a and 1604b. The first interpretation 1604a 
represents the matching of object 1608 of the first frame 1602 to object 
1614 of the second frame 1604, while the second interpretation 1604b 
represents the matching of the object 1608 of the first frame 1602 with 
the object 1612 of the second frame 1604. 

To track the objects from the first frame 1602 to the second 
frame 1604, initially, in operation 1502 each object from the first frame 
1602 is matched with an object in the second frame 1604. For example, 
interpretation 1604a is chosen as the second frame. Object 1608 of the 
first frame 1602 is matched with object 1614 of the second frame 1604. 
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Likewise, object 1610 of the first frame 1602 is matched with object 
1612 of the second frame 1604. Next, in operation 1404, a sum of the 
distances between the paired objects is calculated. For interpretation 
1604a, the sum is equal to distance dl plus d2. For example, distance 
dl represents the distance that object 1608 would have moved if object 
1608 is identified as object 1614. 

After the sum of distances between matched objects is calculated 
for one set of paired objects or interpretation (e.g., 1604a or 1604b), it 
is determined whether any other combinations of paired objects is 
possible in operation 1506. That is, it is determined whether it is 
possible to match each of the objects of the first frame to different 
objects in the second frame. If it is possible to match different objects, 
then the operation 1502 is repeated for a different interpretation (e.g., 
1604b) of the second frame. For instance, object 1608 of the first 
frame 1602 is matched with the object 1612 of the second frame 1604. 
Next, in operation 1404, a second sum is calculated for the new matched 
combinations as shown in interpretation 1604b (e.g., d3 plus d4). 

If it is determined that it is not possible to match each of the 
objects of the first frame to different objects of the second frame, the 
process 1506 proceed to operation 1508. In the example of Fig. 15, no 
other possible matches are possible, so operation 1508 is executed. In 
operation 1508, it is determined which combination results in a 
minimum sum. In other words, it is determined which combination of 
matched pairs results in the least amount of distance moved. As shown 
in Fig. 16, when object 1608 is matched with object 1612 of the second 
frame 1604 and object 1610 is matched with the object 1614 of the 
second frame 1604, the sum is equal to d3 plus d4 as shown in 
interpretation 1604b. This sum is, of course, less than the sum of the 
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distances (dl plus d2) if object 1608 were matched with object 1614 of 
the second frame 1604a and object 1610 were matched with object 1612 
of the second frame 1604a. Thus, it is concluded that object 1608 of the 
first frame 1602 is the same as object 1612 of the second frame, and has 
moved distance dl. This tracking method has the advantage of being a 
fast and effective way of tracking non-coded objects. 

Although the foregoing invention has been described in some 
detail for purposes of clarity of understanding, it will be apparent that 
certain changes and modifications may be practiced within the scope of 
the appended claims. For example, the techniques for tracking coded 
objects of the present invention may be implemented by comparing 
RGB values, rather than luminosity values. By way of another example, 
the techniques for tracking non-coded objects may compare sizes or 
colors of objects of a first frame to sizes or colors of objects of a second 
frame. Accordingly, the present embodiments are to be considered as 
illustrative and not restrictive, and the invention is not to be limited to 
the details given herein, but may be modified within the scope and 
equivalents of the appended claims. 
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CLAIMS 



What is claimed is: 



1 . A method of tracking coded objects that are located within 
5 a physical system and are represented within an image, the method 

comprising the acts of: 

i 

receiving a first image having a plurality of image pixels that 
include a first pixel, a first neighbor pixel, and a plurality of coded 
objects pixels that include the first pixel, wherein the coded object pixels 
1 0 represent a first coded object of the physical system that has a known 

code pattern; 

obtaining object data about the first coded object by comparing 
luminosity values of the first pixel and the first neighbor pixel. 



15 2. The method recited in claim 1, wherein the object data 

specifies a pixel of the plurality of pixels as being a reference pixel 
within the first coded object that has the known code pattern. 



3. The method recited in claim 2, wherein the act of obtaining 
20 the object data is further comprised of the acts of: 

comparing luminosity values of the first pixel and a first group of 
neighbor pixels of the plurality of pixels; 

determining a first count of the first group neighbor pixels that 
have a luminosity value that is lower than a first pixel luminosity value 
25 of the first pixel; 
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determining whether the first count is more than a predetermined 
number; and 

defining the first pixel as the reference pixel of the first code 
object when the first count is more than the predetermined number. 

4. The method recited in claim 2, wherein the act of obtaining 
the object data is further comprised of the acts of: 

comparing luminosity values of the first pixel and a first group of 
neighbor pixels of the plurality of pixels; 

determining a first count of the first group neighbor pixels that 
have a luminosity value that is lower than a first pixel luminosity value 
of the first pixel; 

comparing luminosity values of a second pixel and a second group 
of neighbor pixels of the plurality of pixels; 

determining a second count of the second group neighbor pixels 
that have a luminosity value that is lower than a second pixel luminosity 
value of the second pixel; 

determining whether the second count is more than a 
predetermined number; 

defining the first pixel as the reference pixel of the first code 
object when the first count is higher than the second count; and 

defining the second pixel as the reference pixel of the first code 
object when the second count is higher than the first count. 
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5. The method recited in claim 1, wherein the object data 
indicates a position of the first pixel. 



6. The method recited in claim 5, wherein the act of obtaining 
5 the object data is further comprised of the acts of: 

comparing luminosity values of a first region of pixels of the 
plurality of pixels to determine a minimum value position of a selected 
pixel of the first region that has a minimum luminosity value that is 
lower than any other pixel of the first region, wherein the region 
1 0 includes the first pixel, the first pixel having a black color and the other 

pixels of the region having a white color; and 

defining the position of the first pixel as being at the minimum 
value position within the first region. 



15 7. The method recited in claim 5, wherein the act of obtaining 

the object data is further comprised of the acts of: 

comparing luminosity values of a first region of pixels of the 
plurality of pixels to determine a maximum value position of a selected 
pixel of the first region that has a maximum luminosity value that is 
20 higher than any other pixel of the first region, wherein the region 

includes the first pixel, the first pixel having a white color and the other 
pixels of the region having a black color; and 

defining the position of the first pixel as being at the maximum 
value position within the first region. 
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8. The method recited in claim 5, wherein the object data 
further indicates a position of a second pixel, the first pixel being black 
and the second pixel being white. 

9. The method recited in claim 8, wherein the act of obtaining 
the object data is further comprised of the acts of: 

subtracting the luminosity value of each pixel of the plurality of 
pixels that is positioned within a defined white region from an 
associated pixel of the plurality of pixels that is positioned in a defined 
black region, the pixels of the defined white region being mostly white 
except for the first pixel and the pixels of the defined black region being 
mostly black accept for the second pixel; 

determining a minimum subtraction position for the pixels that 
result in a lowest subtraction result; and 

defining the position of the first pixel and the second pixel as 
being at the minimum subtraction position within the white region and 
the black region. 

10. The method recited in claim 5, wherein the position of the 
first pixel indicates the position of the first coded object. 

11. The method recited in claim 5, wherein the position of the 
first pixel indicates the position of a movable object within the first 
coded object. 



INT1P012/LVP/MRO 



-43 - 



PATENT 



12. The method recited in claim 1, wherein the object data 
indicates whether the first pixel is black or white. 

13. The method recited in claim 12, wherein the first neighbor 
5 pixel is a known white pixel and the object data is obtained by the acts 

of: 

determining whether a first pixel luminosity value of the first 
pixel is smaller than a first neighbor luminosity value of the first 
neighbor pixel by more than a predetermined value; 

10 defining the first pixel as black when the first pixel luminosity 

value of the first pixel is smaller than the first neighbor luminosity 
value of the first neighbor pixel by more than a predetermined value; 
and 

defining the first pixel as white when the first pixel luminosity 
1 5 value of the first pixel is not smaller than the first neighbor luminosity 

value of the first neighbor pixel by more than a predetermined value. 

14. The method recited in claim 12, wherein the act of 
obtaining the object data is further comprised of the acts of: 

20 comparing luminosity values of the first pixel and a second 

neighbor pixel of the plurality of pixels, wherein the first neighbor 
pixel and the second neighbor pixel are known white pixels; and 

determining whether the first pixel luminosity value of the first 
pixel is smaller than a maximum value luminosity of the first neighbor 
25 pixel and the second neighbor pixel by more than a predetermined 

value. 
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15. The method recited in claim 14, wherein the act of 
obtaining the object data is further comprised of the acts of: 

defining the first pixel as black when the first pixel is smaller 
than a maximum value luminosity of the first neighbor pixel and the 
second neighbor pixel by more than a predetermined value; and 

defining the first pixel as white when the first pixel luminosity 
value of the first pixel is not smaller than a maximum value luminosity 
of the first neighbor pixel and the second neighbor pixel by more than a 
predetermined value. 

16. The method recited in claim 14, wherein the act of 
obtaining the object data is further comprised of the acts of: 

determining whether a first pixel luminosity value of the first 
pixel is smaller than a minimum value luminosity of the first neighbor 
pixel and the second neighbor pixel by more than zero; and 

defining the first pixel as white when the first pixel luminosity 
value of the first pixel is smaller than a maximum value luminosity of 
the first neighbor pixel and the second neighbor pixel by more than the 
predetermined value; and 

defining the first pixel as black when the first pixel luminosity 
value of the first pixel is not smaller than a maximum value luminosity 
of the first neighbor pixel and the second neighbor pixel by more than 
the predetermined value and when the first pixel luminosity value of the 
first pixel is smaller than the minimum value by more than zero. 
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17. The method recited in claim 12, wherein the act of 
obtaining the object data is further comprised of the acts of: 

comparing luminosity values of the first pixel and a second 
neighbor pixel of the plurality of pixels, wherein the first neighbor 
5 pixel is a known white pixel and the second neighbor pixel is a known 

black pixel; and 

determining whether a first pixel luminosity value of the first 
pixel is smaller than a first neighbor luminosity value of the first 
neighbor pixel by more than a second neighbor luminosity value the 
1 0 second neighbor pixel is smaller than the first pixel luminosity value of 

the first pixel; 

defining the first pixel as black when the first pixel luminosity 
value of the first pixel is smaller than the first neighbor luminosity 
value of the first neighbor pixel by more than the second neighbor 
1 5 luminosity value of the second neighbor pixel is smaller than the first 

pixel luminosity value of the first pixel; and 

defining the first pixel as white when the first pixel luminosity 
value of the first pixel is not smaller than the first neighbor luminosity 
value of the first neighbor pixel by more than the second neighbor 
20 luminosity value of the second neighbor pixel is smaller than the first 

pixel luminosity value of the first pixel. 

18. The method recited in claim 12, wherein the first neighbor 
pixel has a color that is opposite a color of the first pixel and the object 

25 information is obtained by the acts of: 
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determining whether a first neighbor luminosity value of the first 
neighbor pixel is greater than a first pixel luminosity value of the first 
pixel; 

defining the first pixel as black when the first neighbor 
luminosity value of the first neighbor pixel is greater than the first pixel 
luminosity value of the first pixel; and 

defining the first pixel as white when the first neighbor 
luminosity value of the first neighbor pixel is not greater than the first 
pixel luminosity value of the first pixel. 

19. The method recited in claim 1, wherein the first image 
includes a first non-coded object and a second non-coded object, the 
method further comprising the acts of: 

receiving a second image that includes a third non-coded object 
and a fourth non-coded object, wherein the first image represents a first 
frame at a first point in time and the second image represents a second 
frame of a second point in time; 

calculating a first sum of the distances between the first object of 
the first frame and the third object of the second frame and between the 
second object of the first frame and the fourth object of the second 
frame; 

calculating a second sum of the distances between the first object 
of the first frame and the fourth object of the second frame and between 
the second object of the first frame and the third object of the second 
frame; 

identifying the first object as being equivalent to the third object 
of the second frame when the first sum is less than the second sum; and 
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identifying the first object as being equivalent to the fourth object 
of the second frame when the second sum is less than the first sum; 



20. A coded object that is configured to be scanned by a camera 
5 and included in an image that is received as input to a computer 

implemented coded object recognition program, the coded object 
comprising: 

a first region that is represented as a first region image in the 
received image; and 

10 a second region that is represented as a second region image in 

the received image, wherein the first region has an opposite color of the 
second region such that the coded object may be matched to a known 
coded pattern by determining whether a certain percentage of pixels 
within the second region image have a higher luminosity value than the 

1 5 first region image. 



21. The coded object recited in claim 20, further comprising a 
third region that is represented as a third region image in the received 
image, wherein the third region has a same color as the second region 
20 such that the coded object may be matched to a known coded pattern by 

further determining whether a certain percentage of pixels within the 
third region image have a higher luminosity value than the first region 
image. 



25 22. The coded object recited in claim 21, further comprising an 

angle region having an angle object that are represented as an angle 
region image having an angle object image in the received image, 
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wherein the angle region is mostly white except for the angle object 
such that a position of the angle object within the angle region may be 
determined by finding a pixel having the lowest luminosity value within 
the angle region image. 

23. The coded object recited in claim 22, wherein the position 
of the angle object relative to the first region indicates the orientation of 
the coded object. 

24. The coded object recited in claim 22, further comprising an 
identification object having an identifier position, wherein the position 
of the angle object relative to the first region indicates the identifier 
position, the identification object having a luminosity value that 
identifies the coded object. 

25. The coded object recited in claim 24, wherein the color of 
the identification object is one of a group consisting of a first color and 
a second color such that when the color is the first color, the coded 
object is of a first object type and when the color is the second color, 
the coded object is of a second object type. 

26. The coded object recited in claim 24, wherein the color of 
the identification object has a first portion and a second portion, the 
first portion having an opposite color of the second portion such that an 
identity of the coded object is determined by ascertaining which of the 
first and second portions is black and which is white. 
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27. The coded object recited in claim 20, further comprising an 
angle region having an angle object that are represented as an angle 
region image having an angle object image in the received image, 
wherein the angle object has a dark portion and a light portion and the 
5 angle region has a first half that is mostly white except for the dark 

portion of the angle object and a second half that is mostly black except 
for the light portion of the angle object such that a position of the angle 
object within the angle region may be determined by subtracting each 
pixel of the first half from each pixel in the second half and finding a 
1 0 minimum result. 



28. The coded object recited in claim 27, further comprising an 
identification object having an identifier position, wherein the position 
of the angle object indicates the identifier position, the identification 

1 5 object having a color that identifies the coded object. 

29. The coded object recited in claim 28, wherein the color of 
the identification object is one of a group consisting of a first color and 
a second color such that when the color is the first color, the coded 

20 object is of a first object type and when the color is the second color, 

the coded object is of a second object type. 



30. The coded object recited in claim 28, wherein the color of 
the identification object has a first portion and a second portion, the 
25 first portion having an opposite color of the second portion such that an 

identity of the coded object is determined by ascertaining which of the 
first and second portions is black and which is white. 
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ABSTRACT 



Disclosed is a method of tracking coded objects that are located 
within a physical system and are represented within an image. The 
method includes the act of receiving a first image having a plurality of 
image pixels that include a first pixel, a first neighbor pixel, and a 
plurality of coded objects pixels that include the first pixel, wherein the 
coded object pixels represent a first coded object of the physical system 
that has a known code pattern and the act of obtaining object data about 
the first coded object by comparing luminosity values of the first pixel 
and the first neighbor pixel. In one embodiment, the object data also 
indicates whether the first pixel is black or white, and the first neighbor 
pixel is a known white pixel. The first neighbor pixel has a color that is 
opposite a color of the first pixel, and the act of obtaining the object 
data further includes the acts of determining whether a first pixel 
luminosity value of the first pixel is smaller than a first neighbor 
luminosity value of the first neighbor pixel by more than a 
predetermined value, defining the first pixel as black when the first 
pixel luminosity value of the first pixel is smaller than the first neighbor 
luminosity value of the first neighbor pixel by more than a 
predetermined value, determining whether the first neighbor luminosity 
value of the first neighbor pixel is smaller than the first pixel luminosity 
value of the first pixel by more than a predetermined value when the 
first pixel luminosity value of the first pixel is not smaller than the first 
neighbor luminosity value of the first neighbor pixel by more than a 
predetermined value, and defining the first pixel as white when the first 
neighbor luminosity value of the first neighbor pixel is smaller than the 
first pixel luminosity value of the first pixel by more than a 
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predetermined value. Also disclosed is a coded object that is configured 
to be scanned by a camera and included in an image that is received as 
input to a computer implemented coded object recognition program. 
The coded object includes a first region that is represented as a first 
5 region image in the received image and a second region that is 

represented as a second region image in the received image. The first 
region has an opposite color of the second region such that the coded 
object may be matched to a known coded pattern by determining 
whether a certain percentage of pixels within the second region image 
1 0 have a higher luminosity value than the first region image. 
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